Introduction

The roots of employee (aka work; e.g., Schaufeli & Bakker, 2010) engagement research likely started with theoretical expansions of forms of employee participation and job involvement (e.g., Elloy, Everett, & Flynn, 1991). Like many other constructs within the I-O psychology field, our knowledge of engagement has been at least partially informed by its measurement and documented association with other work attitudes and behaviors. There are currently many engagement scales used for either academic and/or applied purposes (for example, the UWES, ISA, or Q12).

The current engagement survey is being developed within a larger study, with Schaufeli et el.’s (2002) vigor-dedication-absorption structure forming the substantive element structure: engagement is a mental state wherein employees: a) feel energized (Vigor), b) are enthusiastic about the content of their work and the things they do (Dedication), and c) are so immersed in their work activities that time seems compressed (Absorption). Additionally, these three components are each measured at three attitudinal levels: d) feeling (e.g., affect), e) thought (e.g., cognition), and f) action (e.g., behavior).

This study examines the impact of item ordering on factor structure using this measure that is intentionally saturated with two different structural components: 1) attitudinal, and 2) substantive.

Methods

There were two samples used for this study. The first was a snowball sample initiated by IO Psychology faculty and graduate students, which resulted in a total of 282 participants. There were four counterbalanced orderings of item presentations within this administration, as well as an additional 18 contextual items. In the second data collection initiative, Qualtrics panels were solicited. These US workforce representative adults responded to two counterbalanced orderings of the focal 18 items along with 2 additional contextual items. The respondents of the second sample included 343 who responded to attitudinally clustered items and 404 who responded to substantively clustered items.

Results

Two omnibus confirmatory factor analyses (CFAs) were imposed on the data conforming to the two scale definitions reported in Table 1. Regardless of item ordering, across 1,025 respondents, both models showed fair fit (\(χ^2_{substantive}\)=995.34, df=132, RMSEA=0.09; \(χ^2_{attitudinal}\)=1,103.47, df=132, RMSEA=0.09). Additional fit indexes for the two models (as well as models run separately within each condition) are presented in Table 2.Figures 1 and 2 are visual representations of the omnibus CFA models. Table 3 presents unit-weighted scale scores across the three substantive and three attitudinal scales. Omnibus data α’s (across all conditions) were 0.81 (Absorption), 0.91 (Dedication), 0.78 (Vigor), 0.78 (Affect), 0.89 (Cognition), and 0.83 (Behavior).

The effects expected from the six types of item presentations did emerge in the anticipated direction in all conditions, but were quite moderate (for example, \(Δχ^2_{Cond1}\) = 9.55, ΔAIC_Cond1 = 10.53). MEasurement invariance was evaluated across a series of progressively greater restrictive conditions. Because our six conditions were realized across two qualitatively different sampling procedures, we applied our analyses of measurement invariance twice - once within our snowball sample which resulted in strong/scalar invariance for the attitudinal structure and mariganlly attaining scalar invariance for the substantive CFA (see Tables 4 and 5). Secondly, the follow-up Qualtrics panel respondents there was evidence of weak/metric invariance for the substantive CFA across the two Qualtrics item orderings (\(ΔΧ^2\) = 13.84, df = 15, p = .54), but a lack of invariance for the attitudinal model.

Table 1: Summary fit indices across item ordering conditions

Condition	Model	X	Df	RMSEA	SRMR	CFI	TLI	AIC
Condition 1	3-factor substantive	300.86	132	0.14	0.11	0.68	0.63	3,282.88
	3-factor attitudinal	290.33	132	0.14	0.11	0.70	0.65	3,272.35
Condition 2	3-factor substantive	310.01	132	0.15	0.10	0.71	0.66	3,257.45
	3-factor attitudinal	322.52	132	0.15	0.10	0.69	0.64	3,269.96
Condition 3	3-factor substantive	252.07	132	0.12	0.10	0.78	0.74	3,510.32
	3-factor attitudinal	275.74	132	0.13	0.10	0.73	0.69	3,534.00
Condition 4	3-factor substantive	224.96	132	0.10	0.09	0.82	0.79	3,421.64
	3-factor attitudinal	228.99	132	0.10	0.09	0.81	0.78	3,425.66
Condition 5	3-factor substantive	549.8	132	0.10	0.05	0.90	0.89	14,932.57
	3-factor attitudinal	497.9	132	0.10	0.04	0.92	0.90	14,880.67
Condition 6	3-factor substantive	468.02	132	0.09	0.05	0.91	0.90	17,953.02
	3-factor attitudinal	610.6	132	0.10	0.06	0.88	0.86	18,095.61
Overall	3-factor substantive	995.34	132	0.09	0.05	0.91	0.90	46,915.05
	3-factor attitudinal	1,103.47	132	0.09	0.05	0.90	0.88	47,023.18

Discussion

When items are presented “together” in terms of conceptual cohesion (either grouped by attitudinal dimension or substantive dimension), the CFAs yielded slightly better fit indicators than when items were not presented in conceptually cohesive groupings. This was true across different levels of cohesive emphasis. There is evidence of at least partial invariance within both administrations. However, the idea of invariance has greater support within the initial four administration conditions than with the second sampling method.

The practical implication is that item ordering perhaps does not matter so much. We were in a unique circumstance regarding the opportunity to manipulate different meaningful groupings because our focal measure is intentionally complex. The item ordering literature does fairly consistently claim better reliability of assessment scores later in a sequence of similar items, and this was not evaluated in the current study, but could be the focus of future investigations.